The major stages in the life cycle of software development are the design phase, the coding phase, the code complete phase, the alpha phase, the beta phase, and finally, release to market. During the design phase, the customer problems the software product will address and the functionality of the software product is defined. Typically, the completion of the functional specification marks the end of the design phase. The coding phase may already have begun. The code complete phase is reached when the code has been written but is not necessarily debugged. The alpha phase marks the point in time when the product is stable; that is, most of the major bugs have been found. In the beta phase, the product is ideally free of all major bugs; the only bugs remaining should be essentially harmless. When the product passes a final quality assurance checklist, it is ready for release to market.
As no one wants software that does not work, testing is an important part of the life cycle and can span several phases. Software testing involves devising a test case (or, more likely, a set of test cases), running the software with the test case as input, and checking that the performance of the software with the test case as input yields the expected results. Software testing can be conducted manually by humans or programmatically, referred to as automated software testing. Ideally, testing of the software should begin as soon as possible in the life cycle of the software. Generally, however, the software cannot be tested at all until the design phase has been completed, because until the design phase is complete, expected results cannot be determined. Typically, during the coding phase, the developer manually tests his code as he writes it. Automated software testing usually cannot begin until far later in the development process.
Sometimes, the only testing that is conducted is done by the developer who manually tests as he codes. A developer who tests his own work, however, is likely to overlook bugs that someone not so emotionally invested in the code will find. Furthermore, the scope of the developer's testing is typically limited to the functionality of his code and integration of his code with a limited number of other software applications.
To address these shortcomings, many software development houses have a separate software testing group that also tests the software, often using at least partially-automated testing techniques. Typically, the testing group tests complex interactions across features and across applications by writing and running test cases. It is generally agreed that involving the testing group early in the product life cycle, even as early as the design phase, reaps many benefits, including identification of inconsistencies in the functional specification, identification of hard-to-test areas and others. In general, however, the effort required to keep each test case current in the face of continued changes in feature definition, implementation and user interface (UI) tuning renders this approach impractical. Hence, writing and running test cases is typically a hurried matter that occurs at the tail end of product development. Testing and in particular, automated testing, thus tends to be perpetually behind the curve. It would be helpful if there were a way to write test cases and employ automated testing as soon as possible in the life cycle of a software product, ideally during the design phase.
Development of a suite of test cases is a challenge whenever it occurs. To test a specific feature of an application, numerous sets of tests must be written. For example, an application may permit many modes of interaction with a feature: via a mouse, keyboard, digitizer, accessibility software, programmatically, and so on. Therefore, to provide a comprehensive test for the feature, a suite of tests should include a set of tests interacting with the feature via the mouse (typing text just like a user might); one set interacting with the feature via keyboard, one set interacting with the feature via digitizer, one set interacting with the feature via accessibility software to invoke default actions and otherwise mimic an accessibility application, one set interacting with the feature via the application's coding model, and so on. It would be helpful if there were a way to make sure that the suite of test cases produced provided a comprehensive test of the feature or application and further, to decrease the total number of test cases that must be written to provide that comprehensive test.
Furthermore, much or all of the logic in each of these sets of test is identical to the logic in the other sets of tests and typically, much or all of the verification of results processing is identical as well. Hence, many tests are identical or very nearly so, merely varying execution options. For example, for all the multiple forms of input described above, the expected results are likely identical. Hence, writing a test case for each of these input sources typically requires writing a separate method for executing the test for each of the input sources, and duplicating most of the rest of the test script. Writing the same test over and over again with minor variations is tedious and time-consuming. It would be helpful if there were a way to eliminate or significantly reduce this duplicative coding and to reduce the total number of test cases that must be written.
Code written to determine if the actual results of running the test case coincide with the expected results (called verification of results, or verification) is often included within the test case. Changing the details of a particular result verification or adding new result verification typically requires the modification of each test case. It would be helpful if verification code were separate from the test case, making the test case easier to understand, and the verification code easier to reuse and to maintain.
Execution details are often hard-coded into the test case, requiring the design phase to be complete before the test case is written. It would be helpful if there were a way to define test cases in terms of user actions rather than in terms of specific execution details so that test cases could be written earlier in the software development life cycle.
Testing a software application is a crucial step in the initial development of the application and in the implementation of modifications to the application. Software application developers exert much effort in the testing phase of the application's development. Such testing helps ensure that the application responds in an expected manner to a specific action.
Typically, a test case is coded specifically for testing a component of an application. For example, a user-interface (UI) test case may contain code representing the identification of text boxes, buttons, menus, and the like, whereas an application programming interface (API) test case may communicate directly with an API being tested. Writing a test case for each of these methods of execution may require writing primarily the same test case, with the only variation between the test cases being the manner of execution. That is, one test case may specify testing a component of an application using buttons, another may specify testing the component using an object model, and another using keyboard input. Aside from the execution method, these test cases and the expected results of their implementation may be identical.
Additionally, a test case may require the execution of multiple operations, with each operation capable of being completed using any of a number of execution methods (e.g., invoking buttons or menus with a mouse, invoking buttons or menus with a keyboard, invoking a menu with a mouse and a button with a keyboard, etc.). The number of test cases necessary to test all possible combinations of execution methods for completing multiple operations of an application may be large, even for a relatively simple application.
Developing such a large volume of test cases to test applications by all execution methods is a daunting prospect. Such a task could require writing numerous, virtually identical test cases with variation in only the execution method. Also, test case maintenance can be even more labor-intensive and time-consuming than test case creation. When an application is altered, the applicable test cases may require updating to ensure continued compatibility. Updating such a large number of test cases written for each execution method would likewise be a daunting task.
Additionally, some methods of execution may not be tested at all in a test case. The test case may be designed to determine if an application does a particular action without causing unexpected results. The test case may not be concerned with the method of execution of actions that may seem tangential to the purpose of the test case. It would be advantageous, however, if even the tangential steps of a test case were tested by various execution methods even when the focus of the test may be on some other aspect of the application. If the test case is primarily intended, for example, to test an application's ability to draw a blue rectangle, the test case may include a step for opening a new document. Because opening a new document may be tangential to the purpose of the test case, the test case may provide for the action to be completed using a mouse only. It would be useful, however, to test all execution methods for opening a new document because, for example, it may be discovered that opening a new document using a keyboard causes the blue rectangle to in fact be colored red, created as a circle, or otherwise vary from the expected result.
Therefore, there is a need to test applications using various execution methods without requiring virtually identical test cases to be written for each execution method or every combination of possible execution methods.